Computer readable recording medium recording image processing program and image processing apparatus

ABSTRACT

Displayed region size data indicating a size of a screen of a display device, or a size of a region in which an image of a virtual space is displayed on the screen, is obtained. Distance data indicating a distance between a user and the display device is obtained. A position and an angle of view of the virtual camera in the virtual space are set based on the displayed region size data and the distance data.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. application Ser. No.12/352,716, filed Jan. 13, 2009, which claims the benefit of JapanesePatent Application No. 2008-228608, filed Sep. 5, 2008, the contents ofall which are herein incorporated by reference.

FIELD

The technology herein relates to a computer readable recording mediumrecording an image processing program that is executed by a computer ofan image processing apparatus that outputs an image of a virtual spacecaptured by a virtual camera to a display device, and the imageprocessing apparatus. More particularly, the technology herein relatesto settings of the virtual camera in the virtual space.

BACKGROUND AND SUMMARY

Conventionally, a game apparatus is known that captures an image of anobject in a virtual space using a virtual camera and displays the imageon a display device (e.g., Japanese Patent Laid-Open Publication Nos.2002-163676 and 2001-149643). In the game apparatus disclosed inJapanese Patent Laid-Open Publication No. 2002-163676, a determinationboundary box including a plurality of objects in a game space iscalculated, and it is determined whether or not the determinationboundary box falls within the screen in accordance with settings at thattime of the virtual camera. When the settings of the virtual camera donot cause the determination boundary box to fall within the screen, atleast one of a position, an orientation, and an angle of view of thevirtual camera is changed so as to cause the determination boundary boxto fall within the screen.

Also, in the game apparatus disclosed in Japanese Patent Laid-OpenPublication No. 2001-149643, when an enemy character exists within apredetermined distance range from a position of a player character, avirtual camera control means is used to calculate a position and anorientation of the virtual camera that allow the player character tofollow the enemy character and exist within the sight of the player,based on positions and orientations of an enemy object and a playerobject. Thereafter, the virtual camera is set, and an image of a sightviewed from the virtual camera is generated.

However, game apparatuses as described above conventionally have thefollowing problems. In the process disclosed in each of theaforementioned patent documents, the position and the angle of view ofthe virtual camera are set with reference to the position or the like ofan object in the virtual space. Specifically, the position or the likeof the virtual camera is determined so that predetermined objects (e.g.,both a player object and an enemy object) fall within the single screen.For example, it is assumed that an image of three cubic models iscaptured by the virtual camera and is displayed. Firstly, it is assumedthat the virtual space and the real space have the same scale (the sizeof a cube in the real space and the size of a cubic model in the virtualspace have a ratio of 1:1). As shown in FIG. 18A, it is also assumedthat the position and the angle of view of the virtual camera are set sothat all the three cubes are displayed within the screen. In FIG. 18A, adistance from the cubes to the virtual camera is assumed to be 1 m. As aresult, all the three cubes are displayed on the screen as shown in FIG.18B.

On the other hand, it is assumed that, in the real space, the player islocated a distance away from a television having the screen as shown inFIG. 19. In FIG. 19, the distance between the television and the playeris assumed to be 3 m. It is also assumed that the cubes actually existin the real space. For example, the screen of the television isconsidered as a “window”, and the cubes are assumed to actually existimmediately behind the window. Specifically, FIG. 20 shows a positionalrelationship among the player, the television and the cubes when it isassumed that the cubes actually exist. In this case, a region (visualangle) that is viewed by the player through a window (television) is aregion that is indicated by a dashed line 901 of FIG. 20. Therefore, aregion 902 is a region that is normally not viewed from the player. Inother words, the player can normally view only one middle cube that islocated behind the window (television).

However, even in the case of a positional relationship as shown in FIG.20, the three cubes are displayed on the screen as show in FIG. 18B(i.e., the three cubes are seen in the window). When the three cubesapparently fall within the single screen as viewed from the position ofthe player of FIG. 20, this displayed image of the three cubes isunnatural as compared to an image that would be seen if the cubes werein the real space. Therefore, when it is assumed that the three cubesactually exist behind the television in the real space, the displayedimage is unnatural for an image that would be seen from the position ofthe player. Such an unnatural image causes the player to feel thatsomething is wrong, i.e., the image is less realistic.

Therefore, an object certain example embodiments provide a computerreadable recording medium recording an image processing program and animage processing apparatus capable of achieving a more realisticexpression.

Certain example embodiments have the following features to attain theobject mentioned above. Note that reference numerals, additionaldescriptions and the like inside parentheses in this section indicatecorrespondence to embodiments described below for the sake of easyunderstanding, and do not limit the present invention.

A first aspect of certain example embodiments is directed to a computerreadable recording medium recording an image processing programexecutable by a computer of an image processing apparatus for outputtingan image of a virtual space captured by a virtual camera to a displaydevice (2). The program causes the computer to function as a displayedregion size obtaining means (S1), a distance obtaining means (S5), and avirtual camera setting means (S6 to S8). The displayed region sizeobtaining means obtains displayed region size data indicating a size ofa screen of the display device, or a size of a region in which the imageof the virtual space is displayed on the screen. The distance obtainingmeans obtains distance data indicating a distance between a user and thedisplay device. The virtual camera setting means sets a position and anangle of view of the virtual camera in the virtual space based on thedisplayed region size data and the distance data.

Thus, according to the first aspect, it is possible to prevent the imageof the virtual space displayed on the display device from beingunnatural for an image that is seen from a position of the player. As aresult, a more realistic virtual space can be displayed as if it existedat a position where the display device is located (or in the displaydevice). Thereby, it is possible to substantially prevent the user fromfeeling that something is wrong.

In a second aspect based on the first aspect, the displayed region sizedata includes data indicating a width in a predetermined first directionand a width in a second direction perpendicular to the first direction,of the screen of the display device or the region in which the image ofthe virtual space is displayed on the screen. The virtual camera settingmeans sets a horizontal angle of view and a vertical angle of view ofthe virtual camera, based on the first-direction width and thesecond-direction width included in the displayed region size data.

Thus, according to the second aspect, the horizontal angle of view andthe vertical angle of view of the virtual camera are determined based ona vertical width and a horizontal width on the screen of the region inwhich the image of the virtual space is displayed. Thereby, a morerealistic image can be displayed.

In a third aspect based on the first aspect, the displayed region sizedata includes data indicating a width in a predetermined first directionand a width in a second direction perpendicular to the first direction,of the screen of the display device or the region in which the image ofthe virtual space is displayed on the screen. The virtual camera settingmeans sets a horizontal angle of view and a vertical angle of view ofthe virtual camera, based on the first-direction width or thesecond-direction width included in the displayed region size data.

Thus, according to the third aspect, the horizontal angle of view andthe vertical angle of view of the virtual camera are determined based ona vertical width or a horizontal width on the screen of the region inwhich the image of the virtual space is displayed. Thereby, a morerealistic image can be displayed.

In a fourth aspect based on the second aspect, the virtual camerasetting means sets the horizontal angle of view and/or the verticalangle of view of the virtual camera to be larger as the first-directionwidth and/or the second-direction width increase.

In a fifth aspect based on the third aspect, the virtual camera settingmeans sets the horizontal angle of view and/or the vertical angle ofview of the virtual camera to be larger as the first-direction widthand/or the second-direction width increase.

Thus, according to the fourth and fifth aspects, the angle of view ofthe virtual camera can be adjusted, depending on a size of the screen ofthe display device or the region in which the image of the virtual spaceis displayed. Thereby, a more realistic image can be displayed.

In a sixth aspect based on the second aspect, the virtual camera settingmeans sets the horizontal angle of view and/or the vertical angle ofview to be smaller as the distance indicated by the distance dataincreases.

In a seventh aspect based on the third aspect, the virtual camerasetting means sets the horizontal angle of view and/or the verticalangle of view to be smaller as the distance indicated by the distancedata increases.

Thus, according to the sixth and seventh aspects, even when the userhimself or herself moves, so that the distance between the user and thedisplay device is changed, a highly realistic image corresponding to thedistance can be displayed.

In an eighth aspect based on the first aspect, the virtual camerasetting means sets the angle of view of the virtual camera to be largeras the screen of the display device or the region in which the image ofthe virtual space is displayed on the screen increases.

Thus, according to the eighth aspect, the angle of view of the virtualcamera can be adjusted, depending on a size of the screen of the displaydevice or the region in which the image of the virtual space isdisplayed. Thereby, a more realistic image can be displayed.

In a ninth aspect based on the first aspect, the virtual camera settingmeans sets the angle of view to be smaller as the distance indicated bythe distance data increases.

Thus, according to the ninth aspect, even when the user himself orherself moves, so that the distance between the user and the displaydevice is changed, a highly realistic image corresponding to thedistance can be displayed.

In a tenth aspect based on the first aspect, the distance obtainingmeans includes a captured image data obtaining means (S5) and a distancecalculating means (S5). The captured image data obtaining means obtainscaptured image data output from an input device (7) including an imagecapturing means for capturing an image of at least one imaging subjectwhich is placed in a vicinity of the display device. The distancecalculating means calculates a distance between the input device and theimaging subject as a distance between the user and the display device,based on the imaging subject shown in the captured image indicated bythe captured image data.

Thus, according to the tenth aspect, the distance can be obtained moreaccurately, so that a more realistic image can be displayed.

In an eleventh aspect based on the tenth aspect, the distancecalculating means calculates the distance between the input device andthe imaging subject based on a size of the imaging subject shown in thecaptured image.

In a twelfth aspect based on the eleventh aspect, the distance betweenthe input device and the imaging subject that is calculated by thedistance calculating means has a smaller value as the size of theimaging subject shown in the captured image increases.

In a thirteenth aspect based on the tenth aspect, the distancecalculating means calculates the distance between the input device andthe imaging subject based on a distance between a plurality of imagingsubjects shown in the captured image.

In a fourteenth aspect based on the thirteenth aspect, the distancebetween the input device and the imaging subject that is calculated bythe distance calculating means has a smaller value as a distance betweena plurality of imaging subjects increases.

Thus, according to the eleventh to fourteenth aspects, the distancebetween the imaging subject and the input device can be calculated by aneasier process, so that the efficiency of a process by a computer can beincreased.

In a fifteenth aspect based on the first aspect, the virtual camerasetting means, when an object exists in a space to be displayed on thescreen of the display device or the region in which the image of thevirtual space is displayed on the screen, in the virtual space,determines the position and the angle of view of the virtual camera withreference to a position of the object.

Thus, according to the fifteenth aspect, the virtual camera can beplaced, in the virtual space, at a position that is located the distancebetween the user and the television in the real space away from theposition of the object. Thereby, a less unnatural image can be provided.

In the sixteenth aspect based on the first aspect, the virtual camerasetting means, when a plurality of objects exist in a space to bedisplayed on the screen of the display device or the region in which theimage of the virtual space is displayed on the screen, in the virtualspace, determines the position and the angle of view of the virtualcamera with reference to a position of an object closest to the virtualcamera.

Thus, according to the sixteenth aspect, a less unnatural and highlyrealistic image can be displayed as if the virtual space actuallyexisted immediately behind the screen of the display device.

A seventeenth aspect of certain example embodiments is directed to animage processing apparatus for outputting an image of a virtual spacecaptured by a virtual camera to a display device (2) including adisplayed region size obtaining means (10), a distance obtaining means(10), and a virtual camera setting means (10). The displayed region sizeobtaining means obtains displayed region size data indicating a size ofa screen of the display device, or a size of a region in which the imageof the virtual space is displayed on the screen. The distance obtainingmeans obtains distance data indicating a distance between a user and thedisplay device. The virtual camera setting means sets a position and anangle of view of the virtual camera in the virtual space based on thedisplayed region size data and the distance data.

Thus, according to the seventeenth aspect, an effect similar to that ofthe first aspect can be obtained.

Thus, according to certain example embodiments, a less unnatural andhighly realistic image of the virtual space as viewed from the user canbe displayed, thereby making it possible to substantially prevent theuser from feeling that something is wrong.

These and other objects, features, aspects and advantages of certainexample embodiments will become more apparent from the followingdescription when taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external view of a game system 1 according to anembodiment;

FIG. 2 is a functional block diagram of a game apparatus main body 3;

FIG. 3 is a perspective view showing a controller 7 of FIG. 1 as viewedfrom the top and the rear;

FIG. 4 is a perspective view showing the controller 7 of FIG. 3 asviewed from the bottom and the front;

FIG. 5 is a perspective view of the controller 7 of FIG. 3 where anupper housing thereof is cut away;

FIG. 6 is a perspective view of the controller 7 of FIG. 3 where a lowerhousing thereof is cut away;

FIG. 7 is a block diagram showing a configuration of the controller 7 ofFIG. 3;

FIG. 8 is a diagram showing an exemplary captured image;

FIG. 9 is a diagram for describing a game operation using the controller7;

FIGS. 10A and 10B are diagrams for describing a displayed region;

FIGS. 11A and 11B are diagrams for describing an outline of a process inthe embodiment;

FIGS. 12A and 12B are diagrams for describing the outline of the processin the embodiment;

FIG. 13 is a diagram showing a memory map of a main memory 12 of thegame apparatus main body 3;

FIG. 14 is a flowchart showing an image process executed by the gameapparatus main body 3 in detail.

FIGS. 15A and 15B are diagrams for describing a process in step S5 ofFIG. 13;

FIG. 16 is a diagram for describing a process in step S7 of FIG. 13;

FIGS. 17A to 17C are diagrams for describing a process when a pluralityof 3D models are displayed on a screen;

FIGS. 18A and 18B are diagrams showing a relationship between a virtualcamera and cubic models in a virtual space;

FIG. 19 is a diagram showing a positional relationship between a playerand a television in a real space; and

FIG. 20 is a diagram showing a relationship between an angle of view ofa virtual camera and a visual angle of a player.

DESCRIPTION

Hereinafter, example embodiments will be described with reference to theaccompanying drawings. Note that the present invention is not limited tothese examples.

Whole Configuration of Game System

A game system 1 including a game apparatus according to an embodimentwill be described with reference to FIG. 1. FIG. 1 is an external viewof the game system 1. Hereinafter, the game apparatus of this embodimentand a game program will be described, where the game apparatus is astationary game apparatus as an example. In FIG. 1, the game system 1includes a television set (hereinafter simply referred to as a“television”) 2, the game apparatus main body 3, an optical disc 4, acontroller 7, and a marker unit 8. In the game system 1, the gameapparatus main body 3 executes a game process based on a game operationusing the controller 7.

The optical disc 4, which is an exemplary information storing mediumchangeable with respect to the game apparatus main body 3, is detachablyloaded into the game apparatus main body 3. A game program that isexecuted in the game apparatus main body 3 is stored on the optical disc4. On a front surface of the game apparatus main body 3, a slot throughwhich the optical disc 4 is inserted is provided. The game apparatusmain body 3 executes a game process by reading and executing the gameprogram stored on the optical disc 4 which has been inserted into theslot.

The television 2 as an exemplary display device is connected via aconnection cord to the game apparatus main body 3. The television 2displays a game image that is obtained as a result of the game processexecuted in the game apparatus main body 3. The maker unit 8 is providedin the vicinity of the screen of the television 2 (on an upper side ofthe screen in FIG. 1). The maker unit 8 comprises two markers 8R and 8Lat both ends thereof. Specifically, the marker 8R is one or moreinfrared LEDs that output infrared light toward the front of thetelevision 2 (the same is true of the marker 8L). The maker unit 8 isconnected to the game apparatus main body 3, so that the game apparatusmain body 3 can control ON/OFF of each infrared LED included in themaker unit 8.

The controller 7 is an input device which inputs operation dataindicating an operation performed with respect to the controller 7, tothe game apparatus main body 3. The controller 7 and the game apparatusmain body 3 are connected via wireless communication. In thisembodiment, for example, the Bluetooth® technology is used for wirelesscommunication between the controller 7 and the game apparatus main body3. Note that, in another embodiment, the controller 7 and the gameapparatus main body 3 may be connected via wired communication.

Internal Configuration of Game Apparatus Main Body 3

Next, an internal configuration of the game apparatus main body 3 willbe described with reference to FIG. 2. FIG. 2 is a block diagram showingthe configuration of the game apparatus main body 3. The game apparatusmain body 3 has a CPU 10, a system LSI 11, an external main memory 12, aROM/RTC 13, a disc drive 14, an AV-IC 15, and the like.

The CPU 10 executes a game program stored on the optical disc 4 toperform a game process, i.e., functions as a game processor. The CPU 10is connected to the system LSI 11. In addition to the CPU 10, theexternal main memory 12, the ROM/RTC 13, the disc drive 14, and theAV-IC 15 are connected to the system LSI 11. The system LSI 11 performsprocesses, such as controlling data transfer between each part connectedthereto, generating an image to be displayed, obtaining data from anexternal apparatus, and the like. An internal configuration of thesystem LSI 11 will be described below. The volatile external main memory12 stores a program, such as a game program read out from the opticaldisc 4, a game program read out from a flash memory 17, or the like, orvarious kinds of data, and is used as a work area or a buffer area forthe CPU 10. The ROM/RTC 13 has a ROM (so-called boot ROM) which stores aprogram for booting the game apparatus main body 3, and a clock circuit(RTC: Real Time Clock) which counts time. The disc drive 14 reads outprogram data, texture data or the like from the optical disc 4, andwrites the read data into an internal main memory 11 e (described below)or the external main memory 12.

The system LSI 11 also includes an input/output processor 11 a, a GPU(Graphics Processor Unit) 11 b, a DSP (Digital Signal Processor) 11 c, aVRAM 11 d, and the internal main memory 11 e. The parts 11 a to 11 e areconnected with each other via an internal bus (not shown).

The GPU 11 b, which is a part of a drawing means, generates an image inaccordance with a graphics command (image drawing command) from the CPU10. More specifically, the GPU 11 b performs a calculation processrequired to display 3D graphics, such as coordinate conversion from 3Dcoordinates to 2D coordinates (preprocess before rendering) or the like,and a final rendering process, such as attaching texture or the like, inaccordance with the graphics command, to generate game image data. Here,the CPU 10 inputs, to the GPU 11 b, an image generating program requiredto generate game image data in addition to the graphics command. TheVRAM 11 d stores data (e.g., polygon data, texture data, etc.) which isrequired by the GPU 11 b to execute the graphics command. When an imageis generated, the GPU 11 b generates image data using data stored in theVRAM 11 d.

The DSP 11 c, which functions as an audio processor, generates audiodata using sound data or sound waveform (tone color) data stored in theinternal main memory 11 e or the external main memory 12. The internalmain memory 11 e also stores a program or various data as with theexternal main memory 12, i.e., is also used as a work area or a bufferarea for the CPU 10.

The image data and audio data thus generated are read out by the AV-IC15. The AV-IC 15 outputs the read image data via an AV connector 16 tothe television 2, and the read audio data to a loudspeaker 2 a includedin the television 2. Thereby, an image is displayed on the television 2while a sound is output from the loudspeaker 2 a.

The input/output processor (I/O processor) 11 a executes datatransmission and reception between parts connected thereto, or downloadsdata from an external apparatus. The input/output processor 11 a isconnected to the flash memory 17, a wireless communications module 18, awireless controller module 19, an extension connector 20, and anexternal memory card connector 21. An antenna 22 is connected to thewireless communications module 18, and an antenna 23 is connected to thewireless controller module 19.

The input/output processor 11 a is connected via the wirelesscommunications module 18 and the antenna 22 to a network, and cancommunicate with other game apparatuses or various servers connected tothe network. The input/output processor 11 a regularly accesses theflash memory 17 to detect the presence or absence of data that needs tobe transmitted to the network. In the case of the presence of the data,the input/output processor 11 a transmits the data via the wirelesscommunications module 18 and the antenna 22 to the network. Theinput/output processor 11 a also receives data transmitted from othergame apparatuses or data downloaded from a download server via thenetwork, the antenna 22, and the wireless communications module 18, andstores the received data into the flash memory 17. The CPU 10 executes agame program to read out the data stored in the flash memory 17 andutilizes the data in the game program. In addition to data communicatedbetween the game apparatus main body 3 and other game apparatuses orvarious servers, save data (result data or intermediate data of a game)of a game played using the game apparatus main body 3 may be stored intothe flash memory 17.

The input/output processor 11 a also receives, via the antenna 23 andthe wireless controller module 19, operation data transmitted from thecontroller 7, and stores (temporarily stores) the operation data into abuffer area of the internal main memory 11 e or the external main memory12.

Also, the extension connector 20 and the external memory card connector21 are connected to the input/output processor 11 a. The extensionconnector 20 is a connector for interface, such as USB or SCSI. When amedium (e.g., an external storage medium, etc.), a peripheral device(e.g., another controller, etc.), or a wired communications connector isconnected to the extension connector 20, communication with a networkcan be performed without using the wireless communications module 18.The external memory card connector 21 is a connector for connecting anexternal storage medium, such as a memory card or the like. For example,the input/output processor 11 a can access an external storage mediumvia the extension connector 20 or the external memory card connector 21to save data or read out data.

The game apparatus main body 3 is provided with a power button 24, areset button 25, and an eject button 26. The power button 24 and thereset button 25 are connected to the system LSI 11. When the powerbutton 24 is turned ON, power is supplied via an AC adaptor (not shown)to each part of the game apparatus main body 3. Also, if the powerbutton 24 is pressed again while the power supply is ON, the gameapparatus main body 3 is transitioned to a low-power standby mode. Evenin this state, the game apparatus main body 3 is energized, so that thegame apparatus main body 3 can always be connected to a network, such asthe Internet or the like. Note that the power supply which is currentlyON can be turned OFF by pressing the power button 24 for a predeterminedtime or more. When the reset button 25 is pressed, the system LSI 11reboots the boot program of the game apparatus main body 3. The ejectbutton 26 is connected to the disc drive 14. When the eject button 26 ispressed, the optical disc 4 is ejected from the disc drive 14.

Next, the controller 7 will be described with reference to FIGS. 3 and4. FIG. 3 is a perspective view of the controller 7 as viewed from thetop and the rear. FIG. 4 is a perspective view of the controller 7 asviewed from the bottom and the front.

In FIGS. 3 and 4, the controller 7 has a housing 71, and an operationsection 72 comprising a plurality of operation buttons provided on asurface of the housing 71. The housing 71 of this example is in theshape of substantially a rectangular parallelepiped where afront-to-rear direction thereof is a longitudinal direction thereof. Thewhole housing 71 has a size which allows an adult and a child to holdthe controller 7 with one hand. The housing 71 is formed by, forexample, plastic molding.

A cross key 72 a is provided on a central portion closer to a frontsurface of an upper surface of the housing 71. The cross key 72 a is afour-direction push switch in the shape of a cross, and has operationportions corresponding to the respective four directions (frontward,rearward, leftward, and rightward), the operation portions beingprovided at the respective projecting pieces of the cross which arearranged at intervals of 90°. Any one of the frontward, rearward,leftward and rightward directions is selected by a player pushing down acorresponding one of the operation portions of the cross key 72 a. Forexample, by a player operating the cross key 72 a, a movement directionof a player character or the like appearing in a virtual game world canbe designated, or one can be selected and designated from a plurality ofoptions.

Note that the cross key 72 a is an operation section which outputs anoperation signal, depending on the aforementioned direction inputoperation by a player, or may be an operation section of otherembodiments. For example, an operation section may be provided in whichfour push switches are disposed in cross directions, and an operationsignal is output, depending on a push switch pushed down by the player.In addition to the four push switches, a center switch may be providedat an intersection position of the cross directions, i.e., a complexoperation section comprising the four push switches and the centerswitch may be provided. Also, instead of the cross key 72 a, anoperation section may be provided which outputs an operation signal,depending on a tilt direction of a tiltable stick (so-called joystick)which projects from the upper surface of the housing 71. Also, insteadof the cross key 72 a, an operation section may be provided whichoutputs an operation signal, depending on a slide direction of andisc-like member which can be moved in a horizontal direction. Also, atouch pad may be provided instead of the cross key 72 a.

A plurality of operation buttons 72 b to 72 g are provided at the rearof the cross key 72 a on the upper surface of the housing 71. Theoperation buttons 72 b to 72 g are operation sections which outputoperation signals assigned to the respective operation buttons 72 b to72 g when a player pushes down the head portions of the respectivebuttons. For example, functions of a 1st button, a 2nd button, an Abutton, and the like are assigned to the operation buttons 72 b to 72 d.Functions of a minus switch, a home switch, a plus button, and the likeare assigned to the operation buttons 72 e to 72 g. The operationbuttons 72 a to 72 g are assigned the respective functions, depending ona game program executed by the game apparatus main body 3. Note that, inthe exemplary arrangement of FIG. 3, the operation buttons 72 b to 72 dare aligned on a central portion in a front-to-rear direction of theupper surface of the housing 71. The operation buttons 72 e to 72 g arealigned in a lateral direction on the upper surface of the housing 71and between the operation buttons 72 b and 72 d. The operation button 72f is a button of a type whose upper surface is buried below the uppersurface of the housing 71 so that the player is prevented fromunintentionally and erroneously pushing down the button.

Also, an operation button 72 h is provided at the front of the cross key72 a on the upper surface of the housing 71. The operation button 72 his a power supply switch which remotely switches ON/OFF a power supplyfor the game apparatus main body 3. The operation button 72 h is also abutton of the type whose upper surface is buried below the upper surfaceof the housing 71 so that a player is prevented from unintentionally anderroneously pushing down the button.

A plurality of LEDs 702 are provided at the rear of the operation button72 c on the upper surface of the housing 71. Here, the controller 7 isassigned controller identification (number) so as to distinguish it fromother controllers 7. For example, the LEDs 702 are used so as to notifya player of controller identification currently set for the controller7. Specifically, when transmission data is transmitted from thecontroller 7 to the game apparatus main body 3, one of the LEDs 702 isturned ON, depending on the controller identification.

Also, sound holes through which sound is emitted from a loudspeaker (aloudspeaker 706 of FIG. 5) described below to the outside are formedbetween the operation button 72 b and the operation buttons 72 e to 72 gon the upper surface of the housing 71.

On the other hand, a hollow portion is formed on a lower surface of thehousing 71. The hollow portion on the lower surface of the housing 71 isformed at a position where the index finger or the middle finger of aplayer is placed when the player holds the controller 7 with one handwhile directing the front surface of the controller 7 toward the markers8L and 8R. An operation button 72 i is provided on a slope surface ofthe hollow portion. The operation button 72 i is an operation sectionwhich functions as, for example, a B button.

An image capturing device 743 which is a part of the image captureinformation computing section 74 is provided on a front surface of thehousing 71. Here, the image capture information computing section 74 isa system for analyzing image data captured by the controller 7 todetermine a place having a high luminance in the image data and detect acenter-of-gravity position, a size or the like of the place. The imagecapture information computing section 74 has, for example, a maximumsampling cycle of about 200 frames/sec, and therefore, can track andanalyze relatively high-speed movement of the controller 7. A detailedstructure of the image capture information computing section 74 will bedescribed below. A connector 73 is provided on a rear surface of thehousing 71. The connector 73 is, for example, an edge connector which isutilized so as to engage and connect the controller 7 with a connectioncable.

Here, in order to specifically describe certain example embodiments, acoordinate system which is set with respect to the controller 7 isdefined as follows. As illustrated in FIGS. 3 and 4, X, Y and Z axes,which are orthogonal to each other, are defined with respect to thecontroller 7. Specifically, a front-to-rear direction of the controller7 (the longitudinal direction of the housing 71) is assumed to be the Zaxis, and a front surface (a surface on which the image captureinformation computing section 74 is provided) direction of thecontroller 7 is assumed to be the positive direction of the Z axis. Avertical direction of the controller 7 is assumed to be the Y axis, anda lower surface (a surface on which the operation button 72 i isprovided) direction of the housing 71 is assumed to be the positivedirection of the Y axis. A lateral direction of the controller 7 isassumed to be the X axis, and a left side surface (a side surfaceillustrated in FIG. 4, but not in FIG. 3) direction of the housing 71 isassumed to be the positive direction of the X axis.

Next, an internal structure of the controller 7 will be described withreference to FIGS. 5 and 6. Note that FIG. 5 is a perspective view ofthe controller 7 (as viewed from a rear surface side thereof) where anupper housing (a portion of the housing 71) is cut away. FIG. 6 is aperspective view of the controller 7 as viewed from the front surfaceside thereof where a lower housing (a portion of the housing 71) is cutaway. FIG. 6 illustrates a perspective view of a base board 700 of FIG.5 as viewed from a bottom surface thereof.

In FIG. 5, the base board 700 is fixed inside the housing 71. On anupper major surface of the base board 700, the operation buttons 72 a to72 h, an acceleration sensor 701, the LEDs 702, an antenna 754, and thelike are provided. These are connected to a microcomputer 751 (see FIGS.6 and 7) via conductors (not shown) formed on the base board 700 and thelike. The microcomputer 751 functions to generate operation button data,depending on the operation button 72 a or the like, as an exemplarybutton data generating means of certain example embodiments. Thismechanism, which is a known technique, is implemented, for example, bythe microcomputer 751 detecting contact/disconnection of a conductor bya switching mechanism, such as a tactile switch or the like, which isprovided under a key top. More specifically, for example, when anoperation button is pushed down to contact a conductor, a current flows.The microcomputer 751 detects the current flow to determine whichoperation button has been pushed down, and generates a signal, dependingon the operation button.

Also, by a radio module 753 (see FIG. 7) and an antenna 754, thecontroller 7 functions as a wireless controller. Note that a quartzoscillator (not shown) is provided in the housing 71, and generates abasic clock for the microcomputer 751 (described below). Also, theloudspeaker 706 and an amplifier 708 are provided on the upper majorsurface of the base board 700. Also, the acceleration sensor 701 isprovided on a left side of the operation button 72 d on the base board700 (i.e., a peripheral portion of the base board 700, but not a centerportion thereof). Therefore, the acceleration sensor 701 can detect anacceleration including a component due to a centrifugal force as well asa change in direction of a gravitational acceleration, depending on arotation of the controller 7 around the longitudinal direction as anaxis. Therefore, by predetermined computation, the game apparatus mainbody 3 or the like can determine the rotation of the controller 7 basedon the detected acceleration data with satisfactory sensitivity.

On the other hand, in FIG. 6, the image capture information computingsection 74 is provided at a front edge on a lower major surface of thebase board 700. The image capture information computing section 74comprises an infrared filter 741, a lens 742, the image capturing device743, and an image processing circuit 744, which are arranged in thisorder from the front of the controller 7, and are attached to the lowermajor surface of the base board 700. The connector 73 is attached to arear edge on the lower major surface of the base board 700. Also, asound IC 707 and the microcomputer 751 are provided on the lower majorsurface of the base board 700. The sound IC 707 is connected to themicrocomputer 751 and the amplifier 708 via conductors formed on thebase board 700 and the like, and outputs an audio signal via theamplifier 708 to the loudspeaker 706, depending on sound datatransmitted from the game apparatus main body 3.

A vibrator 704 is attached onto the lower major surface of the baseboard 700. The vibrator 704 may be, for example, a vibration motor or asolenoid. The vibrator 704 is connected to the microcomputer 751 via aconductor formed on the base board 700 and the like, and its activationis switched ON/OFF, depending on vibration data transmitted from thegame apparatus main body 3. The activation of the vibrator 704 generatesvibration in the controller 7, so that the vibration is transferred to aplayer's hand holding the controller 7, thereby making it possible toachieve a so-called vibration-feature supporting game. Here, since thevibrator 704 is disposed somehow closer to the front of the housing 71,the housing 71 significantly vibrates while the player is holding thecontroller 7, so that the player easily feels vibration.

Next, an internal configuration of the controller 7 will be describedwith reference to FIG. 7. Note that FIG. 7 is a block diagramillustrating the configuration of the controller 7.

In FIG. 7, the controller 7 comprises a communication section 75 inaddition to the operation section 72, the image capture informationcomputing section 74, the acceleration sensor 701, the vibrator 704, theloudspeaker 706, the sound IC 707, and the amplifier 708.

The image capture information computing section 74 includes the infraredfilter 741, the lens 742, the image capturing device 743, and the imageprocessing circuit 744. The infrared filter 741 passes only infraredlight entering from the front of the controller 7. Here, the markers 8Land 8R which are provided in the vicinity of the display screen of thetelevision 2 are infrared LEDs which emit infrared light toward thefront of the television 2. Therefore, by providing the infrared filter741, images of the markers 8L and 8R can be accurately captured. Thelens 742 collects infrared light passing through the infrared filter 741and causes the light to enter the image capturing device 743. The imagecapturing device 743 may be, for example, a solid-state image capturingdevice, such as a CMOS sensor or a CCD, and captures the infrared lightcollected by the lens 742. Therefore, the image capturing device 743captures only infrared light passing through the infrared filter 741 togenerate image data. An image captured by the image capturing device 743is hereinafter referred to as a captured image. The image data generatedby the image capturing device 743 is processed by the image processingcircuit 744. The image processing circuit 744 calculates a position of asubject whose image is to be captured (also referred to as an “imagingsubject”) (the markers 8L and 8R) in the captured image. Hereinafter, amethod of calculating the position of an imaging subject will bedescribed with reference to FIG. 8.

FIG. 8 is a diagram showing an exemplary captured image. In the capturedimage A1 of FIG. 8, an image 8L′ of the marker 8L and an image 8R′ ofthe marker 8R are laterally arranged. When receiving the captured image,the image processing circuit 744 initially calculates the coordinates ofa position of each region in the captured image that matchespredetermined conditions. Here, the predetermined conditions areconditions for finding an image of an imaging subject (a target image).Specifically, the predetermined conditions are such that the region hasa luminance higher than or equal to a predetermined value (a highluminance region) and a size within a predetermined range. Note that anypredetermined conditions that can be used to find an imaging subject maybe employed. In other embodiments, the predetermined conditions mayrelate to the color of an image.

When a position of a target image is calculated, the image processingcircuit 744 initially finds the aforementioned high luminance regions ascandidates for the target image, from the entire region of a capturedimage. This is because a target image appears as a high illuminanceregion in the image data of a captured image. Next, based on a size ofthe high luminance region thus found, the image processing circuit 744determines whether or not the high luminance region is the target image.The captured image may contain an image caused by sunlight through awindow or light of a fluorescent tube in a room in addition to theimages (target images) 8L′ and 8R′ of the two markers 8L and 8R. In thiscase, such an image may appear as a high illuminance region in additionto the images of the markers 8L and 8R. The aforementioned determinationprocess is for distinguishing the images of the markers 8L and 8R(target images) from other images to accurately find the target images.Specifically, in the determination process, it is determined whether ornot a high luminance region thus found has a size within a predeterminedrange. When the high luminance region has a size within thepredetermined range, the high luminance region is determined to be thetarget image. When the size of the high luminance region is not withinthe predetermined range, the high luminance region is determined to bean image other than the target image.

Further, for a high luminance region which is determined to representthe target image as a result of the determination process, the imageprocessing circuit 744 calculates a position of the high luminanceregion. Specifically, a position of the center of gravity of the highluminance region is calculated. Note that the position of the center ofgravity can be calculated with a scale finer than the resolution of theimage capturing element 743. It is here assumed that an image capturedby the image capturing element 743 has a resolution of 126×96, and theposition of the center of gravity can be calculated with a scale of1024×768. In this case, the coordinates of the position of the center ofgravity is represented with integer values in the range of (0, 0) to(1024, 768). Note that the position of the captured image is assumed tobe represented by a coordinate system (xy coordinate system) where theupper left corner of the captured image is the origin, the downwarddirection is the positive direction of the y axis, and the rightwarddirection is the positive direction of the x axis.

As described above, the image processing circuit 744 calculatescoordinates indicating a position of each region which satisfies thepredetermined conditions in the captured image. Note that thecoordinates calculated by the image processing circuit 744 are referredto as marker coordinates. The marker coordinates are coordinates thatindicate a position of an imaging subject in the coordinate system forrepresenting a position on a plane corresponding to the captured image.The image processing circuit 744 outputs the marker coordinates to themicrocomputer 751 of the communication section 75. The data of themarker coordinates is transmitted as operation data to the gameapparatus main body 3 by the microcomputer 751. Since the markercoordinates vary depending on an orientation (attitude) or a position ofthe controller 7 itself, the game apparatus main body 3 can calculatethe orientation or the position of the controller 7. Although theprocesses until marker coordinates are calculated from a captured imageare carried out by the image processing circuit 744 and/or themicrocomputer 751 of the controller 7 in this embodiment, the capturedimage may be transferred to the game apparatus main body 3 and processessimilar to the subsequent processes may be executed by the CPU 10 or thelike of the game apparatus main body 3.

The controller 7 preferably comprises the acceleration sensor 701 whichsenses accelerations along with three axes (x, y and z axes). Thethree-axis acceleration sensor 701 senses linear accelerations in threedirections, i.e., a vertical direction, a lateral direction, and afront-to-rear direction. In another embodiment, the acceleration sensor701 may be a two-axis acceleration detecting means which senses onlylinear accelerations along two axes in the vertical direction and thelateral direction (or other direction pairs), depending on the type of acontrol signal used in a game process. For example, the three- ortwo-axis acceleration sensors 701 may be of a type which is availablefrom Analog Devices, Inc. or STMicroelectronics N.V. The accelerationsensor 701 may be of a capacitance type (capacitance coupling type)based on a technique of MEMS (Micro Electro Mechanical Systems) obtainedby micromachining a silicon material. However, the three- or two-axisacceleration sensor 701 may be implemented using a technique of anexisting acceleration detecting means (e.g., a piezoelectric type or apiezoelectric resistance type) or other appropriate techniques whichwill be developed in the future.

It is known to those skilled in the art that an acceleration detectingmeans as used in the acceleration sensor 701 can sense only anacceleration (linear acceleration) along a straight line correspondingto each axis of the acceleration sensor 701. In other words, a directoutput from the acceleration sensor 701 is a signal indicating a linearacceleration (static or dynamic) along each of the two or three axes.Therefore, the acceleration sensor 701 cannot directly sense physicalcharacteristics, such as a motion along a non-linear path (e.g., an arc,etc.), a rotation, a rotational motion, an angular displacement, a tilt,a position, an attitude, and the like.

However, it can be easily understood by those skilled in the art fromthe description of the present specification that, by a computer, suchas a processor (e.g., the CPU 10) of the game apparatus, a processor(e.g., the microcomputer 751) of the controller 7 or the like, executinga process based on a signal relating to an acceleration output from theacceleration sensor 701, additional information about the controller 7can be estimated or calculated (determined). For example, a process maybe performed by the computer, assuming that the controller 7 having theacceleration sensor 701 is in the static state (i.e., assuming that anacceleration detected by the acceleration sensor 701 is only agravitational acceleration). In this case, if the controller 7 isactually in the static state, it can be determined based on the detectedacceleration whether or not or how much the attitude of the controller 7is tilted with respect to the gravity direction. Specifically, if astate of the acceleration sensor 701 whose detected axis is orientedvertically downward is assumed as a reference, it can be determinedwhether or not the controller 7 is tilted, based on whether or not 1 G(gravitational acceleration) is applied to the acceleration sensor 701,and it can be determined how much the controller is tilted, based on themagnitude of the acceleration detected by the acceleration sensor 701.Also, in the case of a multi-axis acceleration sensor, by subjecting anacceleration signal of each axis to a process, it can be determined inmore detail how much the controller 7 is tilted with respect to thegravity direction. In this case, a processor may perform a process ofcalculating data about a tilt angle of the controller 7 based on theoutput of the acceleration sensor 701. Alternatively, a process ofapproximately estimating the tilt may be performed based on the outputof the acceleration sensor 701 without performing the process ofcalculating the data about the tilt angle. Thus, by using a processor incombination with the acceleration sensor 701, a tilt, an attitude or aposition of the controller 7 can be determined. On the other hand, whenit is assumed that the acceleration sensor 701 is in the dynamic state,an acceleration depending on a motion of the acceleration sensor 701 isdetected in addition to the gravitational acceleration component.Therefore, a motion direction or the like can be determined by removingthe gravitational acceleration component by a predetermined process.Specifically, when the controller 7 comprising the acceleration sensor701 is dynamically accelerated by a user's hand, various motions and/orpositions of the controller 7 can be calculated by processing anacceleration signal generated by the acceleration sensor 701. Note that,even if it is assumed that the acceleration sensor 701 is in the dynamicstate, a tilt with respect to the gravity direction can be determined byremoving an acceleration depending on a motion of the accelerationsensor 701 by a predetermined process. In another example, theacceleration sensor 701 may comprise a built-in signal processing deviceor another type of dedicated processing device for performing a desiredprocess with respect to an acceleration signal output from a built-inacceleration detecting means before outputting a signal to themicrocomputer 751. For example, if the acceleration sensor 701 detects astatic acceleration (e.g., the gravitational acceleration), the built-inor dedicated processing device may convert a sensed acceleration signalinto a tilt angle corresponding thereto (or another preferableparameter).

In another embodiment, as an acceleration sensor for detecting a motionof the controller 7, a gyro-sensor comprising a rotation element, avibration element, or the like may be employed. An exemplary MEMSgyro-sensor used in this embodiment is available from Analog Devices,Inc. As is different from the acceleration sensor 701, the gyro-sensorcan directly sense a rotation (or an angular velocity) about an axis ofat least one gyro-element included therein. Thus, since the gyro-sensorand the acceleration sensor are basically different from each other, oneof them is selected, depending on the individual application, andprocesses performed for output signals from these devices need to bechanged as appropriate.

Specifically, when a tilt or an attitude is calculated using agyro-sensor instead of an acceleration sensor, a significant change isrequired. Specifically, when a gyro-sensor is used, the value of a tiltis initialized during the start of detection. Thereafter, angularacceleration data output from the gyro-sensor is integrated. Next, theamount of a change in tilt is calculated from the initialized tiltvalue. In this case, the calculated tilt has a value corresponding to anangle. On the other hand, when an acceleration sensor is used tocalculate a tilt, the tilt is calculated by comparing the value of acomponent about each axis of a gravitational acceleration with apredetermined reference. Therefore, the calculated tilt can berepresented by a vector, and an absolute direction can be detected bythe acceleration detecting means without initialization. Also, whereas avalue calculated as a tilt is an angle when a gyro-sensor is used, thevalue is a vector when an acceleration sensor is used. Therefore, when agyro-sensor is used instead of an acceleration sensor, the tilt dataneeds to be subjected to predetermined conversion, taking intoconsideration a difference between the two devices. The characteristicsof gyro-sensors as well as the basic difference between the accelerationdetecting means and the gyro-sensor are known to those skilled in theart, and will not be herein described in more detail. Whereasgyro-sensors have an advantage of directly sensing rotation,acceleration sensors generally have an advantage over the gyro-sensor interms of cost effectiveness when the acceleration sensor is applied to acontroller as used in this embodiment.

The communication section 75 comprises the microcomputer 751, the memory752, the radio module 753, and the antenna 754. The microcomputer 751controls the radio module 753 for wirelessly transmitting transmissiondata while using the memory 752 as a memory area during a process. Also,the microcomputer 751 controls operations of the sound IC 707 and thevibrator 704, depending on data from the game apparatus main body 3which is received by the radio module 753 via the antenna 754. The soundIC 707 processes sound data or the like transmitted from the gameapparatus main body 3 via the communication section 75. Also, themicrocomputer 751 activates the vibrator 704, depending on vibrationdata (e.g., a signal for switching ON/OFF the vibrator 704) or the liketransmitted from the game apparatus main body 3 via the communicationsection 75.

An operation signal (key data) from the operation section 72 provided inthe controller 7, acceleration signals (x-, y- and z-axis directionacceleration data; hereinafter simply referred to as acceleration data)from the acceleration sensor 701, and process result data from the imagecapture information computing section 74, are output to themicrocomputer 751. The microcomputer 751 temporarily stores the receiveddata (the key data, the acceleration data, and the process result data),as transmission data to be transmitted to the wireless controller module19, into the memory 752. Here, radio transmission from the communicationsection 75 to the wireless controller module 19 is performed inpredetermined cycles. Since a game is generally processed in units of1/60 sec, the cycle of the radio transmission needs to be shorter than1/60 sec. Specifically, the game processing unit is 16.7 ms ( 1/60 sec),and the transmission interval of the communication section 75 employingBluetooth® is 5 ms. When timing of transmission to the wirelesscontroller module 19 arrives, the microcomputer 751 outputs transmissiondata stored in the memory 752, as a series of pieces of operationalinformation, to the radio module 753. Thereafter, the radio module 753modulates the operational information using a carrier wave having apredetermined frequency and emits the resultant radio signal from theantenna 754, by means of, for example, the Bluetooth® technique.Specifically, the key data from the operation section 72 provided in thecontroller 7, the acceleration data from the acceleration sensor 701,and the process result data from the image capture information computingsection 74 are modulated into a radio signal by the radio module 753,and the radio signal is transmitted from the controller 7. Thereafter,the wireless controller module 19 of the game apparatus main body 3receives the radio signal, and the game apparatus main body 3demodulates or decodes the radio signal, thereby obtaining a series ofpieces of operational information (the key data, the acceleration data,and the process result data). Thereafter, the CPU 10 of the gameapparatus main body 3 performs a game process based on the obtainedoperational information and a game program. Note that, when thecommunication section 75 is configured using the Bluetooth® technique,the communication section 75 can also have a function of receivingtransmission data wirelessly transmitted from other devices.

Here, a game operation using the controller 7 will be described. Whenthe controller 7 is used to play a game in the game system 1, a playerholds the controller 7 with one hand. In this case, as shown in FIG. 9,the player holds the controller 7 while directing the front surface (aside through which light to be captured by the image capture informationcomputing section 74 enters) of the controller 7 toward the markers 8Land 8R. In this state, the player performs a game operation by changinga tilt of the controller 7, a position (designated position) on thescreen pointed or designated by the controller 7, or a distance betweenthe controller 7 and each of the markers 8L and 8R.

Next, an outline of a process executed by the thus-configured gamesystem 1 will be described. Firstly, in this embodiment, an editor gamein which the player creates a 3D model is assumed. Specifically, theplayer uses the controller 7 to designate a desired position withrespect to a sample 3D model to be displayed on the screen and performvarious operations, thereby freely creating or editing the 3D model.

In this embodiment, a position and an angle of view of a virtual cameraare set based on a distance between the player in a real space and thetelevision 2. More specifically, initially, a distance between thecontroller 7 and the marker unit 8 (equivalent to a distance between theplayer and the television 2) is calculated. Thereafter, in the virtualspace, the virtual camera is placed at a position that is located thecalculated distance away from a 3D model whose image is to be captured.Thereafter, the angle of view of the virtual camera is determined basedon the distance and a size of a displayed region of the screen in whichthe 3D model is to be displayed.

As used herein, the term “displayed region” refers to a region fordisplaying the 3D model in the real space (in other words, a region thatis on the screen in the real space and in which a so-called renderingtarget is displayed). For example, when a 3D model 101 is displayed onthe entire screen as shown in FIG. 10A, the entire screen of thetelevision 2 is a displayed region, and a size (actual size) of thescreen is a size of the displayed region. Alternatively, when awindow-type display is utilized to provide only a partial region of thescreen as a region for displaying a 3D model as shown in FIG. 10B, awindow 102 is a displayed region as used herein, and a size in the realspace of a partial region of the screen corresponding to a size of thewindow 102 (this window is displayed) is a size of the displayed region.

An outline of a process in this embodiment will be described,illustrating an example in which the virtual camera captures an image ofthe 3D model 101 (e.g., a cubic model (hereinafter referred to as a“cubic model 101”; 101 is a reference numeral)) placed in the virtualspace, and the captured image is displayed on the entire screen of thetelevision 2, assuming that the virtual space and the real space havethe same scale (the ratio of a size of a cube in the real space to asize of the cubic model in the virtual space is 1:1). Initially, asshown in FIG. 11A, a distance d in the real space between the television2 and the controller 7 is calculated. Next, as shown in FIG. 11B, in thevirtual space, the virtual camera is placed at a position that islocated the distance d away from the cubic model 101. For example, ifthe distance d between the television 2 and the controller 7 is 3 m, thevirtual camera is also placed, in the virtual space, at a position thatis located 3 m away from the cubic model 101 (a surface thereof facingthe player). Since the real space and the virtual space have the samescale, the distance between the television 2 and the controller 7 is thesame as the distance between the cubic model 101 and the virtual camera.In other words, the position of the virtual camera corresponds to theposition of the player (the controller 7), and the position of the cubicmodel 101 corresponds to the position of the television 2.

Next, the angle of view θ of the virtual camera is set, depending on thedistance d and the size of the displayed region (here, the screen of thetelevision 2). The angle of view θ is decreased with an increase in thedistance d between the television 2 and the controller 7 as shown inFIG. 12A. In other words, as the distance d between the television 2 andthe controller 7 is decreased as shown in FIG. 12B, the angle of view θis increased. Specifically, as the player approaches the screen, theangle of view θ of the virtual camera is set to be wider. Conversely, asthe player goes away from the screen, the angle of view θ of the virtualcamera is set to be narrower.

Thus, the position and the angle of view θ of the virtual camera are setbased on the distance in the real space between the player and thetelevision 2 before an image of the cubic model 101 is captured, therebymaking it possible to display a more realistic image on the screen.Specifically, the same image as that which would be seen by the playerif the cubic model 101 actually existed in the television 2 (orimmediately behind the screen of the television 2) in the real space, isdisplayed on the screen. As a result, it is possible to provide an imagethat does not cause the player to feel that something is wrong.

Next, an image process executed by the game apparatus main body 3 willbe described in detail. Firstly, data that is stored into the externalmain memory 12 during the image process will be described. FIG. 13 is adiagram showing a memory map of the main memory 12 of the game apparatusmain body 3. In FIG. 13, the external main memory 12 includes a programstoring area 120 and a data storing area 122. Data in the programstoring area 120 and the data storing area 122 is stored on the opticaldisc 4, and is transferred to and stored into the external main memory12 during execution of an image processing program.

The program storing area 120 stores, for example, an image processingprogram 121 that is executed by the CPU 10.

The data storing area 122 stores, for example, data, such as operationdata 123, displayed region size data 124, distance data 125, virtualcamera data 126, object data 127, and the like.

The operation data 123 is operation data that is successivelytransmitted from the controller 7, and is updated with the latestoperation data. The operation data 123 includes first coordinate data1231 and second coordinate data 1232. The first coordinate data 1231 iscoordinate data indicating a position of one of the images of the twomarkers 8L and 8R with respect to the image captured by the imagecapturing device 743 (a position in the captured image; here,corresponding to 8L′ in FIG. 8). The second coordinate data 1232 iscoordinate data indicating a position of the image of the other marker(a position in the captured image; here, corresponding to 8R′ in FIG.8). For example, the positions of the marker images are represented byan XY coordinate system in the captured image.

The displayed region size data 124 is data indicating a size of adisplayed region as described with reference to FIG. 10. For example, avertical width and a horizontal width of a displayed region, which arerepresented in units of centimeters or the like, are stored as thedisplayed region size data 124. In this embodiment, data indicating thehorizontal width is described as a horizontal width wx, and dataindicating the vertical width is described as a vertical width wy. Notethat the displayed region size data 124 may be a screen size representedby a diagonal length (e.g., “32 type”, “32 inch”, etc.). The data may beobtained by the player previously entering a screen size of thetelevision 2 using a “main body setting screen” of the game apparatusmain body 3, for example, and may be then stored into the flash memory17. The data may be read and transferred from the flash memory 17 to theexternal main memory 12 during execution of the process of thisembodiment.

The distance data 125 is data indicating the distance d from the markers8L and 8R to the controller 7, which is calculated based on the firstcoordinate data 1231 and the second coordinate data 1232.

The virtual camera data 126 is data indicating a position and the likeof the virtual camera in the virtual space. The virtual camera data 126includes camera attitude data 1261, camera position data 1262, andcamera angle-of-view data 1263. The camera attitude data 1261 is dataindicating an attitude of the virtual camera (an orientation of thevirtual camera). The attitude of the virtual camera is represented bythree-dimensional vectors correspond to three axes X, Y and Z (athree-dimensional vector is herein represented by <an X-axis component,a Y-axis component, a Z-axis component> (e.g., <1, 0, 1>)).Specifically, the attitude of the virtual camera is represented by athree-dimensional vector <Zx, Zy, Zz> indicating an orientation in aZ-axis direction, a three-dimensional vector <Yx, Yy, Yz> indicating anorientation in a Y-axis direction, and a three-dimensional vector <Xx,Xy, Xz> indicated an orientation in an X-axis direction (the coordinatespace is assumed to be a right-handed coordinate system).

The camera position data 1262 is data indicating a position of thevirtual camera in the virtual space. In this embodiment, the position ofthe virtual camera is represented by three-dimensional coordinates (cx,cy, cz). The camera angle-of-view data 1263 is data indicating an angleof view of the virtual camera, including a horizontal angle of view rxand a vertical angle of view ry. These values are calculated in aprocess described below.

The object data 127 is data indicating a 3D model placed in the virtualspace. For example, the object data 127 is data indicating the cubicmodel 101 of FIGS. 11A and 11B.

Next, an image process executed by the game apparatus main body 3 willbe described with reference to FIGS. 14 to 16. Note that FIG. 14 is aflowchart showing the image process executed by the game apparatus mainbody 3 in detail. Note that, in the flowchart of FIG. 14, a process ofsetting the virtual camera that is performed by executing the imageprocessing program will be mainly described, and other processes that donot directly relate to certain example embodiments will not be describedin detail.

In FIG. 14, initially, the CPU 10 obtains data indicating the size of adisplayed region (step S1). Specifically, the displayed region size data124 is read out from the external main memory 12 to obtain preset data,i.e., the horizontal width wx and the vertical width wy. Note that whenthe displayed region size is not previously set, the CPU 10 may displaya predetermined input screen to prompt the player to input a screensize. When the displayed region size data 124 is represented by adiagonal length, such as “32 type”, “32 inch” or the like, thehorizontal width wx and the vertical width wy may be calculated from thediagonal length. It is here assumed that the unit of the size is set tobe centimeter. Here, the unit indicating the displayed region size ispreferably the same as the unit of the distance d described below.Therefore, the distance d described below is also assumed to berepresented in units of centimeters.

Next, the CPU 10 reads out the object data 127 from the external mainmemory 12 and generates a 3D model based on the data. Thereafter, theCPU 10 places the generated 3D model at a predetermined position in thevirtual space (step S2). Hereinafter, the position (three-dimensionalcoordinate point) where the 3D model is placed is represented by (ax,ay, az).

Next, the CPU 10 sets an attitude of the virtual camera (an orientationof the virtual camera) (step S3). The attitude may be any attitude.Here, as an example, the CPU 10 sets the camera attitude data 1261 sothat the shooting direction of the virtual camera is set to be a frontdirection. Specifically, the three-dimensional vector <Zx, Zy, Zz>indicating the Z-axis orientation is set to be <0, 0, 1>, thethree-dimensional vector <Yx, Yy, Yz> indicating the Y-axis orientationis set to be <0, 1, 0>, and the three-dimensional vector <Xx, Xy, Xz>indicating the X-axis orientation is set to be <1, 0, 0>. That is, thelength of each vector is 1.

Next, the CPU 10 sets a distance n from the virtual camera to a nearclipping plane and a distance f from the virtual camera to a farclipping plane (step S4). These distances are set to have sufficientvalues that cause the 3D model placed in step S2 to be included within aview volume.

Next, the CPU 10 calculates the distance d between the controller 7 andthe markers 8L and 8R based on the first coordinate data 1231 and thesecond coordinate data 1232 that have been transmitted from thecontroller 7 and stored in the external main memory 12 (step S5). Theprocess of step S5 will be described in more detail. Initially, the CPU10 obtains the first coordinate data 1231 and the second coordinate data1232, and calculates a distance mi between two points. As shown in FIG.15A, the two-point distance mi is a distance between two points in acaptured image. The two points correspond to the captured images (8L′and 8R′) of the markers 8L and 8R, and their coordinates have beenobtained as the first coordinate data 1231 and the second coordinatedata 1232. Therefore, the CPU 10 can calculate the two-point distance miusing the first coordinate data 1231 and the second coordinate data1232. Specifically, the two-point distance mi is calculated by:

mi=√{square root over ((Rx−Lx)²+(Ry−Ly)²)}{square root over((Rx−Lx)²+(Ry−Ly)²)}

where the first coordinate data 1231 is assumed to be positioncoordinates (Lx, Ly), and the second coordinate data 1232 is assumed tobe position coordinates (Rx, Ry).

Next, the CPU 10 calculates a width w (see FIG. 15B) of a range withinwhich the image capturing device 743 can capture images with respect ofpositions where the markers 8L and 8R are placed. The width w iscalculated by:

w=wi×m/mi

where m represents a spacing between the markers 8L and 8R (actualspacing; e.g., 20 cm) (m has a fixed value), and wi represents a widthof an image captured by the image capturing device 743 corresponding tothe width w (wi has a fixed value). Since the spacing m and the width wieach have a fixed value, they are previously stored in a memory means(e.g., the flash memory 17) in the game apparatus main body 3. Next, theCPU 10 calculates the distance d (see FIG. 15B) between the markers 8Land 8R and the image capturing device 743 (the controller 7) using thewidth w and the angle of view θ of the image capturing device 743, andstores the distance d into the distance data 125. Here, the distance dcan be calculated by:

d=(w/2)/{tan(θ/2)}

where the angle of view θ has a fixed value, and therefore, ispreviously stored in the game apparatus main body 3 (e.g., the flashmemory 17).

Referring back to FIG. 14, next, the CPU 10 calculates a position (cx,cy, cz) that is located the distance d from the 3D model in the virtualspace, and places the virtual camera at the position (step S6). The CPU10 also stores data indicating the position into the camera positiondata 1262. The position (cx, cy, cz) can be calculated by, for example,the following expression.

cx=ax−d×Zx

cy=ay−d×Zy

cz=az−d×Zz

Next, the CPU 10 executes a process of calculating the angle of view ofthe virtual camera, i.e., the horizontal angle of view rx and thevertical angle of view ry (step S7).

The horizontal angle of view rx and the vertical angle of view ry arecalculated by the following expression.

rx=2×ATAN(wx/2/d)

ry=2×ATAN(wy/2/d)

FIG. 16 shows a relationship among the positions of the virtual cameraand the 3D model, the distance d, and the horizontal width wx when thehorizontal angle of view rx is calculated. Here, the horizontal angle ofview rx is set to have a value that is increased with an increase in thehorizontal width wx of the displayed region. Specifically, even when thedistance d has the same value (the position of the player is the same),a television with a screen having an aspect ratio of 4:3 and atelevision with a screen having an aspect ratio of 16:9 have differenthorizontal angles of view rx, i.e., the latter has a larger angle ofview rx, for example. Also, even when the aspect ratio is the same, a19-inch television and a 32-inch television have different angles ofview, i.e., the 32-inch television has a larger angle of view (the sameis true of the vertical angle of view ry).

Referring back to FIG. 14, next, the CPU 10 generates a camera matrixfrom the camera position data 1262, the camera attitude data 1261, andthe virtual camera angle-of-view data 1263 (step S8). The camera matrixincludes a 3×4 view matrix for conversion into a direction in which thevirtual space is viewed from the virtual camera, and a 4×4 projectionmatrix for projection onto a 2D screen. Specifically, the followingmatrix expressions are generated.

3×4 View Matrix Expression

-   -   m00 m01 m02 m03    -   m10 m11 m12 m13    -   m20 m21 m22 m23

m00=−Xx

m01=−Xy

m02=−Xz

m03=cx×Xx+cy×Xy+cz×Xz

m10=Yx

m11=Yy

m12=Yz

m13=−cx×Yx−cy×Yy−cz×Yz

m20=−Zx

m21=−Zy

m22=−Zz

m23=cx×Zx+cy×Zy+cz×Zz

4×4 Projection Matrix Expression

-   -   m00 m01 m02 m03    -   m10 m11 m12 m13    -   m20 m21 m22 m23    -   m30 m31 m32 m33

m00=2×n/(R−L)

m01=0

m02=(R+L)/(R−L)

m03=0

m10=0

m11=2×n/(T−B)

m12=(T+B)/(T−B)

m13=0

m20=0

m21=0

m22=−(f+n)/(f−n)

m23=−2×f×n/(f−n)

m30=0

m31=0

m32=−1

m33=0

R=wx×(n/d)×−0.5

L=wx×(n/d)×0.5

T=wy×(n/d)×0.5

B=wy×(n/d)×−0.5

Next, the CPU 10 executes a drawing process for displaying an image ofthe virtual space in which the 3D model is placed (the image is capturedby the virtual camera set in the aforementioned process) (step S9).

After step S9, the CPU 10 determines whether or not the process is to beended (step S10). If the result of determination is YES, the process isended. If the result of determination is NO, the process flow returns tostep S5, and the process is repeated. Thus, the image process of thisembodiment is completed.

Thus, in this embodiment, a position and an angle of view of a virtualcamera are set, depending on a distance in a real space between a playerand a television. An image captured by the thus-set virtual camera canprovide a more realistic sensation to the player.

Note that it has been assumed in the embodiment above that the distancebetween the television 2 and the player is obtained by using acombination of the controller 7 with an image capturing device and themarker unit 8. It has also been assumed that the distance d iscalculated based on the spacing between the markers 8L and 8R. Thepresent invention is not limited to this. The distance d may becalculated based on a size of a captured image of the marker 8L or 8R.For example, the distance d may be calculated based on a diameter of themarker image 8L′ or 8R′ in a captured image (see FIG. 15A). This isbecause the distance d decreases with an increase in the diameter, i.e.,the distance d increases with a decrease in the diameter.

Although it has been assumed above that the distance between thetelevision 2 and the player is obtained by using a combination of thecontroller 7 with an image capturing device and the marker unit 8, thepresent invention is not limited to this. For example, the player maymeasure a distance between the television 2 and the player and manuallyenter the measured distance. The measured distance may be temporarilystored in a memory, and may be read out from the memory as required. Anymethod that can be used to obtain a distance between a player and atelevision may be employed.

Also, the process of setting the distance n to the near clipping planeand the distance f to the far clipping plane in step S4 may be executedafter the virtual camera is placed in step S6. In this case, by settingthe near clipping plane and the far clipping plane after placing thevirtual camera, the setting can be performed more appropriately.

Also, when a plurality of 3D models are displayed on the screen, thevirtual camera is preferably placed with reference to a 3D model that islocated closest to the player (the virtual camera is focused on theclosest 3D model). For example, it is assumed that two cubic models 101as shown in FIG. 11B are displayed and placed at positions that aredifferent from each other in a depth direction, i.e., one of the twocubic models is placed closer to the player than the other cubic modelis (the other cubic model is placed farther from the player). In thiscase, as shown in FIG. 17A, a position of the 3D model closest to theplayer may be set to be (ax, ay, az) shown in step S2, and the virtualcamera may be placed at a position that is located the distance d awayfrom the position (ax, ay, az). Thereby, the 3D model closest to theplayer is displayed as if it were located behind the screen of thetelevision 2, thereby making it possible to provide a realistic imagewithout unnaturalness. In addition, as shown in FIG. 17B, the virtualcamera may be placed with reference to a position of a 3D model that islocated at a deeper position as viewed from the player. Alternatively,as shown in FIG. 17C, the virtual camera may be placed with reference toa middle position of a distance in a depth direction within which aplurality of 3D models are located. When the virtual camera is placed asshown in FIG. 17B or 17C, a 3D model closer to the player can bedisplayed as if it were located closer to the player than the screen ofthe television 2 is. Also in this case, it is possible to provide arealistic image as if a 3D model were located closer to the player thanthe screen of the television 2 is.

While certain example embodiments have been described in detail, theforegoing description is in all aspects illustrative and notrestrictive. It is understood that numerous other modifications andvariations can be devised without departing from the scope of theinvention.

1. A non-transitory computer readable storage medium storing an imageprocessing program executable by a processing system of an imageprocessing apparatus for outputting an image of a virtual space capturedby a virtual camera to a display device, the program comprisinginstructions that are configured to: obtain distance data indicating adistance between the display device and an input device configured to beheld by one hand of a user; and set a display range of the virtualcamera in the virtual space based on a display region size and thedistance data.
 2. The medium of claim 1, wherein the instructions arefurther configured to: obtain the display region size in which the imageof the virtual space is displayed on a screen, the display region sizeincluding a first width in a first direction and a second width in asecond direction perpendicular to the first direction; set a horizontalangle of view and a vertical angle of view of the virtual camera basedon the first width and the second width.
 3. The medium of claim 1,wherein the instructions are further configured to: obtain the displayregion size in which the image of the virtual space is displayed on ascreen, the display region size including a width in a first directionand a width in a second direction perpendicular to the first directionof the display region; and set a horizontal angle of view and a verticalangle of view of the virtual camera, based on the width in the firstdirection or the width in the second direction.
 4. The medium of claim2, wherein the horizontal angle of view and/or the vertical angle ofview of the virtual camera is set to increase as the first width and/orthe second width increase.
 5. The medium of claim 3, wherein thehorizontal angle of view and/or the vertical angle of view of thevirtual camera is set to increase as the width in the first directionand/or width in the second direction increase.
 6. The medium of claim 2,wherein the horizontal angle of view and/or the vertical angle of viewis set to be smaller as the distance indicated by the distance dataincreases.
 7. The medium of claim 3, wherein the horizontal angle ofview and/or the vertical angle of view is set to be smaller as thedistance indicated by the distance data increases.
 8. The medium ofclaim 1, wherein the display range of the virtual camera is set to belarger as the display region size in which the image of the virtualspace is displayed on the screen increases.
 9. The medium of claim 1,wherein the display range of the virtual camera is set to be smaller asthe distance indicated by the distance data increases.
 10. The medium ofclaim 1, wherein the input device includes an imager that is configuredto capture an image of at least one imaging subject proximate to thedisplay device, the instructions are further configured to: obtaincaptured image data output the input device based on the captured image;and calculate the distance between the input device and the imagingsubject as a distance between the user and the display device based onthe imaging subject shown in the captured image indicated by thecaptured image data.
 11. The medium of claim 10, wherein the distance iscalculated based on a size of the imaging subject shown in the capturedimage.
 12. The medium of claim 11, wherein the calculated distancebetween the input device and the imaging subject decreases as the sizeof the imaging subject shown in the captured image increases.
 13. Themedium of claim 10, wherein the distance between the input device andthe imaging subject is calculated based on a distance between aplurality of imaging subjects shown in the captured image.
 14. Themedium of claim 13, wherein the calculated distance between the inputdevice and the imaging subject decreases as a distance between aplurality of imaging subjects increases.
 15. The medium of claim 1,wherein the display range of the virtual camera is further based on aposition of an object that exists in the virtual space.
 16. The mediumof claim 1, wherein the instructions are further configured to: when aplurality of objects exist in a space to be displayed on the displaydevice, determine the display range of the virtual camera with referenceto a position of an object closest to the virtual camera.
 17. An imageprocessing apparatus for outputting an image of a virtual space capturedby a virtual camera to a display device, comprising: a processing systemthat includes at least one processor, the processing system configuredto: obtain distance data indicating a distance between an input deviceconfigured to be held by one hand of a user and the display device; andset a display range of the virtual camera in the virtual space based ona region size associated with the display device and the distance data.18. An image processing system that is configured to output an image ofa virtual space to a display device, the system comprising: at least oneprocessor that is configured to: obtain distance data that relates to adistance between a handheld controller and the display device; and set adisplay range a virtual camera in the virtual space based on a displayregion size that is associated with the display device and the distancedata.
 19. A computer implemented method for outputting an image of avirtual space to a display device, the method comprising: obtainingdistance data that relates to a distance between a handheld user inputdevice and the display device; and setting a display range a virtualcamera, via a processing system that includes at least one processor,that exists in the virtual space based on the distance data and adisplay region size that is associated with the display device.
 20. Themedium of claim 1, wherein: the display range is set to be decrease asthe distance indicated by the distance data increases, and the displayrange is set to increase as the distance indicated by the distance datadecreases.